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(54) Non-intrusive measurement of end-to-end network properties 



(57) A method and apparatus for non-intrusive 
measurement of end-to-end properties of a network flow 
uses a passive approach. Data units at the input of a 
network path are sampled by an ingress monitor, which 
identifies each data unit with a unique signature and 
generates measurement data. The same data units at 
the output of the network path are sampled by an egress 
monitor, which identifies each data unit with a unique 



signature and generates measurement data. A data cor- 
relator pulls lists of entries from the ingress and egress 
monitors, each entry having the unique signature and 
the measurement data for one of the data units, corre- 
lates the lists to find entries from the lists that have the 
same unique signature, and determines from the meas- 
urement data for the data units from the two lists that 
have the same unique signature the end-to-end proper- 
ties of the network flow. 
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Description 

BACKGROUND OF THE INVENTION 



r0001l The present invention relates to testing of net- 
works, and more particularly to a method and apparatus 
for non-intrusive measurement of end-to-end properties 
of data transmission through a networic path. 
[00021 A network provides uni-directional virtual or 
physical path or channel for data transmission. A net- 
work flow is a session of data transmission through a 
network path. In a network flow data units flow from an 
Ingress to an egress of the network path. Example net- 
works are Internet Protocol (IP) networks, Asynchro- 
nous Transmission Mode (ATM) networks and crcurt- 
switched. such as telephony, networks. Example date 
units are IP packets, ATM cells and data bytes (octets). 
For simplicity the following is described in the environ- 
ment of IP networks, so the terms IP networks, IP virtual 
path, IP packet and packet flow are used. 
r0003] The end-to-end propert.es of a packet flow 
may be measured by several metrics, such as through- 
put, end-to-end latency, packet loss. etc. There .are two 
approaches to measuring the metrics. One is active and 
the other is passive. „„»„ rin n 
r00041 In an active approach packets for monitoring 
purposes, with special Identities and time-stamps, are 
injected into the ingress of a packet flow. At the egress 
these special monitoring packets are identified, re- 
moved from the flow and time-stamped. Metrics, such 
as end-to-end latency and packet loss may be meas- 
ured by analyzing the received monitoring packets^An 
advantage of the active approach is that it is straightfor- 
ward and simple, and may be used either online or off- 
line. The main disadvantage of the active approach 
when used online is that, no matter how small the inject- 
ed monitoring packet flow is, these monitoring packets 
always interfere with the operational packet flow. Also, 
since the network condition is dynamic, off-line meas- 
urement might not reflect the actual properties of the 
packet flow. 

Fo005] In a passive approach no monitonng packets 
are injected into the packet flow being monitored In- 
stead at both the ingress and egress of the packet flow 
the operational data packets are intercepted without in- 
terference to the packet flow itself, and measurement 
data is generated for each data packet. The measure- 
ment data from the two points are correlated, and end- 
to-end property metrics are derived. In contrast to the 
active approach the passive approach is non-intrusive. 
It does not interfere with the operational packet flow in 
any manner. The cost of this benefit is that the passive 
approach is much more complex and harder to imple- 
ment Due to the reliance on the operational packet flow 
the passive approach does not assess the end-to-end 
properties of a network path before its actual operation. 
r0006] A passive approach for ATM network monrtor- 
ng systems is the WAND project by the University of 



Waitato in New Zealand which is disclosed at httpJ/atm. 
cs waikataac.nztoand. WAND focuses on delay meas- 
urement for ATM networks, specifically uses a CRC for 
data correlation (matching of ATM cells), and matches 
5 onecellfromtheingresstoalistofcellsfromtheegress^ 
ra007] What is desired is a non-intrusive method of 
monitoring end-to-end properties of network flows in an 
efficient way. 



10 BRIEF SUMMARY OF THE INVENTION 

I0008] Accordingly the present invention provides a 
method and apparatus . for non-intrusive measurement 
of end-to-end properties of network flows using a pas- 
ts sive approach. An ingress monitor non-intrusively inter- 
cepts data units as they enter a network path through a 
network. Likewise an egress monitor non-intrusively in- 
tercepts the same data units as they leave the same net- 
work path. Each monitor generates a time stamp for 
20 each intercepted dala unit using a common clock and 
derives a unique signature for each data unit such that 
the same data unit has the same signature at the entry 
as at the exit. Additionally each monitor counts the 
number of packets received from the network flow at the 
25 ingress and egress respectively. The signature time 
stamp and packet counter value form an entry which is 
retained in an entry queue in each monitor A d ate co r- 
mlator coupled to an out-of-band network to wh eh the 

monitors also are coupled periodically pulls a list of en- 
30 tries from each monitor and correlates the'isteby 
matching signatures. The time stamps for the entnes 
from ft respective lists having the same signature^ are 
pressed to obtain end-to-end latency of the network 
oath and the difference in counts over a given time pe- 
35 riod provides end-to-end packet loss measurement 
5,009] The objects, advantages and other novel tea- 
kires of the present invention are described in the fol- 
lowing detailed description when read in conjunction 
with the appended claims and attached drawing. 

40 BRIEF DESCRIPTION OFTHE SEVERAL VIEWS OF 
THE DRAWING 

f00101 Fig. 1 is an architecture view of a system for 
45 non-intrusive monitoring of end-to-end properties of a 
network path according to the present invention 
mom Fig. 2 is a flow chart diagram of a method of 
non-intrusive monitoring of end-to-end properties of a 
network path according to the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

[0012] Referring now to Fig. 1 an overall architecture 
of a non-intrusive approach for monitoring end-to-end 
55 properties of a netwoik path is shown. To measure the 
end-to-end properties a monitor 1 0, 20 is placed at ®^h 
of the ingress 12 and egress 22 of a packet flow 14 
through a network 16. These two monitors 10, 20 are 
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coupled to a data correlator 18 through an out-of-band 
network 24 that is independent of and does not interfere 
with the packet flow 14 being monitored. The two mon- 
itors 10. 20 and the data correlator 18 have a common 
clock 26. The two monitors 10, 20 at the two end points 
12, 22 intercept packets from the network flow being 
monitored, count the data packets received, and extract 
measurement data from the packets, time stamping the 
measurements. The data correlator 1 8 pulls the meas- 
urement data from the two monitors 10, 20, correlates 
them to identify the data from the two monitors belong- 
ing to the same packets, and derives measurement re- 
sults. 

[0013] The monitor 10 at the ingress 12 intercepts in 
real time all the data packets flowing into the path 1 4 in 
a non-intrusive wav. When a data packet from the data 
packet flow 14 is intercepted, the monitor 10 generates 
a time-stamp and extracts a signature of the data pack- 
et. The monitor 10 also keeps a count of ail packets re- 
ceived so far. The packet signature is a piece of infor- 
mation to be used to differentiate the data packet in 
question from other data packets in the same packet 
flow 14. In most cases a special form of Cyclic Redun- 
dancy Check (CRC) of the data packet, such as CRC of 
all fields in a data packet except those that may change 
hop by hop In an IP packet header, may be used as the 
packet signature. Other Information, such as some 
fields of the data packet header, may also be extracted 
as needed. As a result for each data packet the monitor 
10 generates an entry composed of <signature, time- 
stamp, count>, i.e., the packet signature and time- 
stamp together with a count of the packets received so 
far. The monitor 1 0 maintains a queue to hold all recent 
entries in the incremental order of the time-stamps for 
retrieval and processing by the data correlator 18. Ex- 
actly how long a period of times entries are held de- 
pends on several factors, such as the end-to-end packet 
flow latency, bandwidth and latency of the out-of-band 
network 24 between the data correlator 1 8 and monitors 
10, 20. 

[0014] The monitor 20 at the egress 22 does exactly 
the same things as the monitor 10 at the ingress 12 on 
the data packets flowing out of the egress. 
[0015] The data correlator 18 samples end-to-end 
property measurements periodically. During such sam- 
pling the data correlator 18 pulls a list of entries from 
each of the two monitors 10, 20, correlates the two lists 
and derives end-to-end packet flow property measure- 
ments. The data correlator 1 8 correlates the two lists by 
matching the packet signatures (S-i; S-e) from the two 
lists until a subset in one list matches a subset in the 
other list. When the correlation is done, if one entry in 
one list matches one entry in the other list, these two 
entries contain the same packet signature and are con- 
sidered to be the measurements for the same data pack- 
et (P-1). The entry from the ingress monitor 10 (P-i-1: 
<S-i-1,T-M,C-i-1>) contains the time-stamp (T-i-1) 
when the data packet enters the packet flow, while the 



entry from the egress monitor 20 (P-e-1 : <S-e-1 , T-e-1 , 
C-e-1>) contains the time-stamp (T-e-1) when the data 
packet leaves the packet flow, where S-i-1 = S-e-1 . The 
difference between these two time-stamps Is the end- 
s to-end latency for the data packet to flow through the 
network path 1 4. The frequency with which the data cor- 
relator 18 pulls data from the monitors 10, 20 and the 
amount of data it pulls each time depends on the specific 
configuration in use, the amount of available bandwidth 
10 of the out-of-band network 24, and other relevant fac- 
tors. 

[0016] At a latertlme n the data correlator 1 8 identifies 
another pair of matched entries (P+n: S-i-n,T-l-n,C-i-n), 
(P-e-n: S-e-n,T-e-n,C-e-n) where S-l-n = S-e-n, then 
w ((C-i-n - C-i-1 ) - (C-e-n - C-e-1 )) is the number of packets 
lost between P-1 and P-n. The packet loss ratio is ((C- 
i-n - C-i-1 ) - (C-e-n-C-e-1 ))/(C-i-n - C-i-1 ) - (C-e-n - C-e- 
1) and the packet loss rate is ((C-i-n - C-i-1) - (C-e-n - 
C-e-1 ))/(T-i-n -T-i-1). 
20 [0017] The following further illustrates the measure- 
ment data correlation. Suppose at time T the data cor- 
relator 18 pulls from the monitors 10, 20 all entries be- 
tween Ts and T, where Ts<T. The list from the ingress 
monitor 10 is: List-i: (P-i-1 :<S-M , T-i-1 ,C-i-1>, P-i- 
25 m:<S-i-m, T-i-m, C-l-m>) where (P-M :<S-i-1 , T-i-1 , C-i- 
1 >) is an entry with packet signature S-I-1 , time-stamp 
T-i-1 and count C-i-1 . The 1 1st from the egress monitor Is 
[0018] Llst-e: (P-e-1 :<S-e-1, T-e-1, C-e-1 >, .... P-e-n: 
<S-e-n, T-e-n, C-e-n>) Suppose that the sub-list (P-i- 
30 x1, P-i-xk) of List-i matches the sub-list (P-e-yL, .... P- 
e-yk) of List-e, i.e., S-i-xj = S-e-yj forj from 1 to k. |n that 
case S-i-xj and S-e-yj are considered to be the packet 
signatures of the same data packet P-j, and T-i-xj and 
T-e-yj are the times when the data packet P-j passed 
35 the ingress 1 2 and egress 22 respectively. Then T-e-yj- 
T-i-xj is the end-to-end latency for the data packetto flow 
through the network path 14. By calculating the meas- 
urements for all the data packets in the matched sub- 
lists, a set of latency measurements is derived. 
40 p)01 9] Similarly the packet loss between any two data 
packets may be calculated from their packet count val- 
ues, as illustrated above. 

[0020] The latency measurements on individual data 
packets generated in the above process may be used 
45 for analysis of various short-term and long-term laten- 
cies of network flows. The latency measurement sam- 
ples taken over a long period of time (hours, days, 
weeks or months) may be used to generate statistics, 
such as percentile, histogram and distribution, for anal- 
so ysis of the long-term distributions and trends in the la- 
tency of network flows. Each list of latency measure- 
ments that the data correlator 18 gets covers a short 
period of time and may be used to derive short-term la- 
tency jitter. By sampling lists of latency measurements 
55 periodically the long-term trends of latency jitter may be 
analyzed. 

[0021] Similarly based on the individual packet loss 
measurements short-term and long-term packet loss 
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properties may be analyzed. 
[0022] In the above discussion for purposes of illus- 
tration it is assumed that the data correlator 1 8 periodi- 
cally pulls measurement packet lists from the Ingress 
and egress monitors 10, 20. In an actual application the 
ingress and egress monitors 1 0,20 may push the meas- 
urement packets to the data correlator 18. In fact the 
push model may be more efficient in some cases. 
[0023] The architecture of Fig.1 is conceptual. In ac- 
tual implementations in order to reduce the bandwidth 
requirements of the out-of band network 24 the data cor- 
relator 1 8 function may be located in one of the two mon- 
itors 10, 20. The combined data processor/monitor im- 
plementation has at least two advantages: (a) there is 
a reduced network bandwidth requirement; and (b) it is 
feasible for the data correlator 1 8 to get a much longer 
list of data from the resident monitor 10, 20, thus in- 
creasing the search space during data correlation. On 
the other hand a physically separate data correlator 18 
may easily be shared by multiple pairs of monitors 10, 
20 monitoring different packet flows. This data correlator 
18 may also serve as a centralized location for meas- 
urement network configuration and management. In the 
case where a large number of packet flows are to be 
monitored simultaneously, more than one data correla- 
tor 1 8 may be needed to distribute the workload among 

[0024] Subsampling of measurement packets may be 
necessary if the monitors 10, 20, the out-of-band net- 
work 24 or the data processor 1 8 cannot capture, trans- 
port or process the measurement data of all the data 
packets in the packet flow. With measurement packet 
subsampling the monitors 10, 20 still count all packets, 
but only generate measurement data of a selective sub- 
set of the data packets in the packet flow, and the end- 
to-end properties are derived from the measurement da- 
ta of this subset of data packets. There are two require- 
ments a subsampling scheme has to satisfy. One re- 
quirement is that the ingress and egress monitors 10, 
20 select the same subset of data packets. Otherwise if 
the two measurement data lists from the two end-points 
12, 22 are for different sets of data packets, it becomes 
impossible to derive the end-to-end properties. The oth- 
er requirement is that the sampling be periodical or ran- 
dom additive based on some statistical distribution. Sig- 
nature bit masking is a subsampling scheme that satis- 
fies both requirements. With signature bit masking a da- 
ta packet is selected only if the least significant part of 
its signature has a specific given value. For example to 
subsample one-eighth of the data packets, select only 
those data packets whose signature has 000 as its least 
significant three bits. If a data packet is selected by the 
ingress monitor 1 0, it also is selected by the egress mon- 
itor 20 since the packet signature does not change in- 
side the packet flow 14. When CRC is used for the sig- 
nature, since CRC is well randomized, signature bit 
masking becomes a very good form of geometric sam- 
pling in which a packet is selected with a fixed probabil- 



ity which is a form of random additive sampling. 
[0025] The uniqueness of the packet signature affects 
the reliability of the data correlation by the data correla- 
tor 18 Different data packets may have the same con- 
5 tent and thus the same CRC. Even data packets wfch 
different contents may have the same CRC. Non- 
uniqueness of packet signatures introduces ambiguity 
in the list correlation. Carefully selecting the fields to be 
included in the special CRC for the packet signature 
10 helps to improve uniqueness. For example since the 
packet identification field in an IP header contains the 
value of a counter which normally increments by one 
upon sending of each IP packet by a host, inclusion of 
this field in the packet signature CRC may make the 
15 packet signatures of multiple packets unique even if 
they contain the same payload. There may be cases 
when there are more than one way the measurement 
data lists from the two monitors 10, 20 match against 
each other. If this ambiguity occurs, there is no way to 
20 tell reliably which match is correct. As a result it is not 
possible to derive latency measurements from the two 
lists in question. Increasing the number of entries in the 
list of measurement data pulled from the monitors 10, 
20 helps to reduce the probability of ambiguity, but re- 
25 q uiresmorebandwidthfromtheout-of-bandnetwork24. 
[0026] Some of the properties of IP networks made 
data correlation more complicated. IP packets may not 
only be dropped, they may also be duplicated or frag- 
mented by the network 16. They also may be delivered 
30 out of order. Depending upon the configuration of the 
network 1 6, some of these problems may not exist. 
[00271 When the present invention is applied to ATM 
networks, some form of CRC may still be used as the 
cell signature. When applied to circuit-switched net- 
35 works in which the data units are bytes, the whole byte 
itself may be used in place of its signature. 
[0028] Thus the present invention provides non-intru- 
sive measurements of end-to-end properties of network 
flows by sampling the data packets at both the ingress 
40 and egress of the networkflow being measured, assign- 
ing a unique signature and time-stamp using a common 
system clockfor each packet as well as a count of pack- 
ets received so far, correlating lists of packets from both 
ingress and egress based on the unique signature, and 
45 using the time-stamps and counts for the same packet 
at ingress and egress obtaining measurements of the 
desired end-to-end properties of the networkflow. 



50 Claims 

1 A method of non-intrusive measurement of end-to- 
end properties of a network flow, the network flow 
being a session of data units flowing from an ingress 
55 to an egress of a network path , comprising the steps 
of: 

sampling data units at the ingress using an in- 
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2. 



gress monitor; 

generating a unique signature and input meas- 
urement data for each sampled data unit at the 
ingress monitor; 

sampling the data units from the network flow 
at the egress using an egress monitor; 
generating the unique signature and output 
measurement data for each sampled data unit 
at the egress monitor; 

correlating entries from each of the ingress and 
egress monitors, the entries being in the form 
of a list for each monitor and each entry having 
the unique signature and associated measure- 
ment data for one of the data units, by matching 
the unique signatures from entries between the 
two lists; and 

determining from the input and output meas- 
urement data for the matched entries the end- 
to-end properties of the network flow. 

The method as recited in claim 1 wherein the input 
and output measurement data are timestamps indi- 
cating the time of receipt of the data units at the in- 
gress and egress monitors, the timestamps being 
derived from a common clock. 
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3. The method as recited In claims 1 or 2 wherein the 
input and output measurement data are counts of 
the numbers of data units received by the ingress 
and egress monitors so far. so 

4. The method as recited in claim 1 further comprising 
the step of subsampling the data units prior to the 
generating steps such that the ingress and egress 
monitors select the same subsampled data units 35 
with a random additive distribution. 



10. 



a data correlator coupled to an out-of-band net- 
work to which the ingress and egress monitors 
are also coupled, the data processor pulling 
lists of entries from the ingress and egress 
monitors, the entries containing the unique sig- 
nature and measurement data for each data 
unit, correlating the lists of entries based on the 
unique signatures, and determining from the In- 
put and output measurement data of data units 
having the same unique signatures the end-to- 
end properties for the network flow. 

The system as recited in claim 6 wherein the input 
and output measurement data are timestamps for 
the data units derived from a common clock indicat- 
ing the time of arrival atthe ingress and egress mon- 
itors respectively. 

The system as recited in claims 5 or 6 wherein the 
input and output measurement data are counts of 
the number of data units received at the ingress and 
egress monitors respectively so far. 

The system as recited in claim 5 wherein the ingress 
and egress monitors subsample the data units prior 
to generating measurement data, the data units se- 
lected being the same at the Ingress and egress 
monitors with a random additive distribution. 

The system as recited in claim 9 wherein the data 
units are subsampled based upon a given value for 
a least significant portion of the unique signature. 



The method as recited in claim 4 wherein the sub- 
sampling step comprises the step of masking of the 
unique signature such that only if the least signifi- 
cant portion of the unique signature has a specific 
given value is the data unit selected by the ingress 
and egress monitors. 



40 



6. A system for non-intrusive measurement of end-to- 45 
end properties of a network flow comprising: 

an ingress monitor coupled to the input of the 
network flow for intercepting data units flowing 
through a network path, the ingress monitor so 
identifying a unique signature and generating 
input measurement data for each data unit; 
an egress monitor coupled to the output of the 
network flow for intercepting the data units flow- 
ing through the network path, the egress mon- 55 
itor identifying the unique signature and gener- 
ating output measurement data for each data 
unit; and 
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(54) Non-intrusive measurement of end-to-end network properties 



(57) A method and apparatus for non-intrusive 
measurement of end-to-end properties of a network flow 
uses a passive approach. Data units at the input of a 
network path are sampled by an ingress monitor, which 
identifies each data unit with a unique signature and 
generates measurement data. The same data units at 
the output of the network path are sampled by an egress 
monitor, which identifies each data unit with a unique 



signature and generates measurement data. A data cor- 
relator pulls lists of entries from the ingress and egress 
monitors, each entry having the unique signature and 
the measurement data for one of the data units, corre- 
lates the lists to find entries from the lists that have the 
same unique signature, and determines from the meas- 
urement data for the data units from the two lists that 
have the same unique signature the end-to-end proper- 
ties of the network flow. 
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